Outcome harvesting is the most participatory of the methods in this chapter, and the one where a machine has least business doing the interesting part. In this paper we ask which of the questions commissioners put to a harvest could be answered better with a machine's help, and which parts of the work should stay with people.

In short#

Outcome harvesting is mostly a conversation between people. We ask which two steps, in the middle of it, a machine should handle at all.

• We started from terms of reference. We took three published harvests: Oxfam Novib's global programme evaluation, NIMD's evaluation across Mali, Mozambique, Tunisia and Colombia, and ActionAid Denmark's Tax Justice programme. Thirteen questions between them, and none asks how much of the change was attributable to the funder.
• A machine can help with two questions. First, do these outcomes form a pattern of progress towards the objectives? We classify each outcome against objectives declared in advance, then count by objective, by actor type and by year. Second, which objectives have nothing against them?
• An evaluator cannot get the answer to the second any other way. Because the objectives are a declared list, an objective without a single outcome against it still appears in the table, as an empty cell.
• People do everything else. The harvest workshop, the drafting loop, substantiation and sense-making are human steps. For those, we use the workflow only to record that they happened and what they changed.
• Drafting for the room. When people cannot all be in a room for as long as the method assumes, a machine can draft a table of candidate outcomes from interview narratives for them to argue over. In the last section we report one such draft and what it leaves out.

See also: Pilots, not passengers — reconstructible reporting in evaluation; Contribution analysis; A simple measure of the goodness of fit of a causal theory to a text corpus.

What commissioners ask a harvest#

We took the questions from the terms of reference of three published harvests. Method guides say what a method is for, which is not always what commissioners buy it for.

Oxfam Novib's global programme evaluation asked four questions, in a harvest Wilson-Grau carried out with Juliette Majot and Wolfgang Richert: to what extent counterparts achieved outcomes and contributed to policy and practice changes; whether the programme responded effectively to a changing global context; whether counterparts added value to Oxfam's campaigning; and "How has GloPro contributed to the achievements of outcomes by counterparts".

NIMD's evaluation across Mali, Mozambique, Tunisia and Colombia asked three: whether the outcomes "represent patterns of progress towards their respective SP and DfS programme objectives", how well the outcomes "match country-level ToCs", and how suitable outcome harvesting was as a method.

ActionAid Denmark's Tax Justice programme asked six, one of which the report says may be unanswerable by an outcome evaluation at all.

None of the thirteen questions asks how much of the change was attributable to the funder. The questions that recur are also narrower than the method's reputation would lead you to expect:

• What changed, in actors we do not control, including what nobody planned? The harvest itself.
• Do these outcomes form a pattern of progress towards the objectives?
• How well do the outcomes match the theory of change?
• How did we contribute to them?
• Was this method any good here?

The second and third are questions of analysis, asked of an outcomes table that already exists. Published guidance says least about them. They are also where most harvests disappoint the people who paid for them. This paper is about those two.

Evaluators refuse questions, in writing#

Refusing a commissioner's question is sometimes treated as a luxury. The evaluators in all three harvests did it in public.

• Oxfam's evaluators rewrote a sub-question from the terms of reference, saying the original wording "is misleading because we agreed in the Approach paper that we would not make a comparison between planned outcomes and the outcomes actually achieved", and that they had therefore corrected it.
• NIMD's evaluators refused a judgement of worth: "we cannot draw conclusions about the significance of what was achieved ... since we are assessing the achievement of programmatic objectives but not the merit or value of those changes."
• ActionAid's report does not make any recommendations. It cites Wilson-Grau and Britt that an outcome harvest evaluator can rarely recommend action because too many determining factors are unknown to them, and offers discussion points instead.

The refusals in these papers therefore have precedent. We state each one in a sentence of the report: what was asked, what we answered instead, and why.

The denominator problem, in the practitioners' own words#

A count from a harvest has no base, because a harvest is a search and nothing in it was sampled. The published reports say so plainly. Most secondary accounts do not.

Oxfam's report states that the outcomes "are not exhaustive. They are a sampling of what the 38 counterparts consider to be amongst their ten most significant outcomes". It adds that "Caution should be taken, however, in making quantitative comparisons and contrasts between outcomes and outcomes that are PPCs" (policy and practice changes), because fewer simple outcomes and more policy changes is not necessarily a better result.

ActionAid's report goes further and shows what can stand in for sampling. Representativeness is claimed through a gap-filling round: at the end of the harvesting workshop, participants looked at the whole set, named change areas they thought were missing or under-represented, and added outcomes in those areas the following week. The report also notes two biases in the resulting counts. Most outcomes are recent, and a first harvest skews large, with only 6 per cent rated minor.

The gap-filling round is outcome harvesting's own version of a coverage check, and the participants do it themselves. A machine can do something close to it: given the outcomes and the objectives, it can list the objectives with nothing against them. It cannot get the people who were there to notice what is missing, as the workshop did.

ActionAid's contribution percentages#

The standard line, which we also take in the contribution analysis paper, is that these methods yield no attribution fraction. ActionAid's harvest produced one.

Change agents were asked to rate their own contribution to each outcome as a percentage. The rating was kept as a formal classification field, beside year, country, social actor and significance. The report includes a figure of outcomes by contribution percentage. Ratings clustered above 60 per cent, with nineteen outcomes at 80 and three at 90. The evaluator then adjusted percentages downwards where the documentary evidence was thin. Substantiators also rated contribution independently: of eighteen substantiated outcomes, they rated only three lower than the change agent's own figure, the largest gap being 90 per cent against 50.

So the orthodoxy is a position practitioners take, and the method itself does not enforce it. The case also shows what such a number is made of: a self-rating by an interested party, adjusted by an evaluator and agreed by a substantiator whom the same party nominated. A commissioner who wants a percentage badly enough will get one like this. The workflow should then record the number's provenance so that nobody can lose it.

Substantiation in practice#

In Oxfam's harvest every other outcome was put to substantiation, 112 in all. Ninety-five independent people responded, covering 66 outcomes: 30 were fully substantiated, 21 partially and 12 mixed. Substantiators disagreed on only three. Forty-six attempts failed because no substantiator could be reached.

There are two ways to read this, and the report does not favour either. Perhaps the outcome descriptions were accurate. Or perhaps substantiation, as practised, rarely overturns anything, because the people whose account is being checked nominate the substantiators. To tell the two apart we would need to know how often a substantiator nominated by somebody else disagrees. The data do not show that.

For a workflow, the lesson is to record substantiation properly: who nominated the substantiator, what they were asked, and whether the outcome changed as a result. In the record, a substantiation step that never changes anything should be distinguishable from one that had nothing to change.

Who does the work, and at what cost#

With this method the cost of participation is easier to see than with the others.

Oxfam's evaluators corresponded with each counterpart, typically for two or three rounds, until they agreed a title, a description of what changed, when and where, its significance, and the counterpart's contribution. The team budgeted an average of 11.5 hours per counterpart, and reports that some counterparts spent two to four times as long on it as the evaluators did.

In a participatory method the iteration is the method. Most of its cost falls on people who are not being paid to evaluate. Anyone proposing to speed this up with a machine has to say whose time it saves. Drafting outcome statements faster mostly saves the evaluator's.

We propose this division of labour:

• People do the harvest. Who counts as a social actor, what counts as a change, and which ten outcomes matter most are judgements for the people who were there.
• People do the drafting loop. At most, a machine can check that each draft has its four parts and flag those that do not.
• A machine helps with the analysis, which is the part that disappoints commissioners.
• People do the sense-making. A pattern a machine finds is a proposal for the workshop to accept or reject.

Solid boxes are steps people take; dashed boxes are the machine's. Rubicon handles two steps in the middle of what is otherwise a conversation.

The two questions the workflow answers#

We picked these two because the other questions are for the harvest itself or for a workshop.

Do these outcomes form a pattern of progress towards the objectives? We code the outcomes table, classifying each outcome against the objectives declared in advance, then count by objective, by actor type and by year. The denominator problem comes up at the counting stage. Every figure therefore states that its base is the set of outcomes harvested, and the report says so once, prominently, in the main text.

Which objectives have nothing against them? This is the first question turned round. An evaluator cannot answer it any other way. Because the objectives are a declared list, an objective without any outcomes against it appears in the table with a count of nought. It is the machine's version of ActionAid's gap-filling round. We would run it before the sense-making workshop, so that the workshop has something to argue with.

Both use ordinary Rubicon steps: a coding step that treats the outcomes as sources, a counting step that groups by a declared column, and a write-up that states its base. The list of objectives can be declared either as one coding pass per objective or as a category column with its values fixed in advance. Declaring the list is what makes the empty cells appear.

Drafting a table for the workshop#

Deciding what changed, and which changes matter, is still a judgement for the people who were there. But you cannot always get everyone into a room for as long as every topic needs, and drafting is a different act from deciding. People who start from a blank page spend their first hour remembering. Give them a few hundred candidate statements taken from the transcripts and they spend that hour disagreeing, which is the work you wanted from them. Some statements will be activities, some will be the same outcome twice, and some will belong to nobody in the room. Striking those out is part of the harvest.

We would use such a table only if it is marked as a draft, shows the passages behind every row, and leaves to the room everything the room was going to decide: which changes are significant, whose contribution counts, and what the set as a whole shows.

The workflow has three steps. Only the wording of one instruction is specific to outcome harvesting. A sampling step takes the sources and records how many it took. In the coding step the model reads them against the method's own definition of an outcome (a change in the behaviour, relationships, activities, policies or practices of an individual, group, community or organisation) and fills in the method's four fields: who changed, what changed, why it matters locally, and how the intervention contributed. A counting step tabulates the rows. These are the four fields a harvest uses, so the draft can go straight into the method without being translated first.

We ran it once, on 5 September 2026, on the app's own example project: nineteen anonymised household interviews from two provinces. It produced 272 outcome rows from all nineteen documents, between four and thirty-two per document, naming 138 distinct actors. Every quotation matched its source document exactly.

The draft lacks most of what the two questions above need.

• No objectives were declared, so it cannot show a gap. An objective with nothing harvested against it is simply absent from the table. Adding the list is most of the work of answering the second question.
• It does not state a base. The figure of 272 is the number of statements one pass found in nineteen documents. It should not be read as 272 outcomes that happened.
• It counts passages. If two interviews describe the same change, that makes two rows. Deciding whether two accounts describe one event is for the people in the room.
• It cannot tell an outcome from an activity. Some of the 272 describe a service being delivered, which outcome harvesting does not treat as an outcome.
• It grouped by the wrong columns. The counts split the outcomes by province and by the respondent's sex, which are properties of the interview. A harvest needs them by objective, actor type and year, which are properties of the outcome. Those columns were coded but never counted.

One column in that run was wrong in every row. The workflow asked the model to write down which interview each outcome came from, while showing it one chunk of text at a time. The model invented 39 different answers for nineteen documents (1, N/A, Document 1 and variations), none of them right. Meanwhile the record already had the correct document against every quotation. The model was asked for a field that was already a fact about the record. On a real evaluation this would be expensive to notice, because the column looks filled in.

What this cannot do#

• It cannot do the harvest. A machine reading documents afterwards cannot replace a room of people deciding what changed.
• It cannot give a count that means anything on its own. A harvest has no denominator, and every figure has to say so.
• It cannot judge worth. NIMD's evaluators refused to, in as many words. In Rubicon a judgement of worth needs a rubric that somebody has registered.
• It cannot substantiate. Substantiation is a person going on the record. In the workflow we can only note that it happened and whether it changed anything.
• The unit problem again. An outcome concerns a social actor, while Rubicon counts documents.

Next steps#

• Run the two questions against a real outcomes table with its objectives. If you have one you could share, we would like to hear from you.
• Put a draft table in front of people who know the setting, and record what they strike out. How much they strike out is the only real measure of whether drafting helped.
• Record substantiation as a first-class thing: who nominated whom, what changed, and what did not.

This page was drafted with the assistance of Claude, an AI model from Anthropic; the arguments and judgements are our own.